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LONG-TERM  GOALS 

My  long  term  goal  is  to  better  understand  ensemble  forecasting  in  general  and  the  dynamics  of 
initial  uncertainty  given  only  imperfect  models  when  predicting  any  physical  system.  Of  particular 
interest  to  me  are  methods  which  aim  to  evaluate  predictions  of  high  dimensional  systems  like  the 
atmosphere,  methods  to  best  combine  different  models,  and  determination  of  aims  which  are  both 
achievable  by  the  forecaster  and  useful  for  the  user. 

OBJECTIVES 

I  wish  to  establish  exactly  how  uncertainty  in  the  initial  condition  and  imperfections  in  the  model 
limit  the  prediction  of  nonlinear  dynamical  systems  (NDS)  like  the  atmosphere  and  ocean.  Of 
particular  interest  is  the  evaluation  and  interpretation  of  ensemble  forecasts,  now  routinely  made  at 
the  European  Centre  for  Medium-Range  Weather  Eorecasting  (ECMWE)  and  the  National  Center 
for  Environmental  Prediction  (NCEP).  Particular  questions  include  (i)  how  to  best  distribute 
computational  resources  between  using  larger  ensembles  versus  higher  resolution  models,  (ii)  when 
to  select  a  best  model  versus  when  to  use  an  ensemble  over  models,  (iii)  how  to  evaluate  current 
ensemble  formation  schemes  and  (iv)  whether  the  answers  to  these  questions  change  when  regional 
forecasts  are  of  interest.  While  the  focus  is  on  geophysical  fluid  dynamics,  the  insights  gained  are 
applicable  across  a  wide  range  of  forecasting  and  control  systems,  including  mechanical  and 
industrial  processes  and  general  questions  of  data  analysis  and  assimilation.. 

APPROACH 

My  initial  approach  is  to  develop  tests  of  internal  consistency  and  then  apply  them  in  the  context  of 
simple  numerical  and  laboratory  systems  where  more  reliable  statistics  can  be  gathered  than  in  the 
operational  systems.  These  initial  applications  often  suggest  improvements  in  technique  and 
understanding  which  ease  application  to  operational  models,  where  my  approach  has  already 
uncovered  several  fundamental  inconsistencies,  both  in  model  formulation  and  evaluation. 

WORK  COMPLETED 

A  general  overview  of  ensemble  forecasting  and  the  limitation  of  current  ensemble  formation 
schemes,  focusing  on  the  perfect  model  scenario,  was  completed  (Smith,  Ziehmaim  and  Fraedrich, 
1999). 

Review  of  the  goals  of  ensemble  forecasting  in  the  face  of  real  world  limitations  was  begun,  and 
alternative  (achievable)  targets  were  identified  to  replace  ideal  (but  unphysical)  targets. 
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Tests  we  developed  were  used  to  evaluate  the  duration  of  the  linear  regime  in  operational  weather 
forecasts,  this  is  a  crucial  parameter  in  determining  the  consistency  of  ensemble  formation 
techniques. 

A  new  variety  of  shadowing  trajectory  was  defined,  and  used  to  show  the  limitations  model  error 
puts  on  data  assimilation  techniques. 

General  conclusions  of  Lorenz  and  Emanuel  (1988)  on  the  relative  merits  of  different  schemes  for 
making  adaptive  observations  (selecting  which  additional  observations  to  take  among  a  wide  range 
of  potential  observations),  were  reviewed  and  modified,  and  then  extended  to  the  imperfect  model 
case.  Methods  to  test  the  relevance  of  our  conclusions  in  an  operational  setting  were  developed 
(Hansen  and  Smith,  1999). 

A  new  cost  function  was  derived  showing  the  limitations  of  the  common  one  step  least  squared 
error  criteria;  at  present  applications  are  limited  to  low  dimensional  systems,  but  both  deterministic 
and  stochastic  models  are  considered  (McSharry  and  Smith,  1999). 

Using  models  suggested  by  Lorenz  (1996),  a  scheme  has  been  developed  to  provide  a  test  bed  for 
exploring  the  improvement  of  the  parameterisation  of  sub-gridscale  processes,  and  for  better 
understanding  the  model  sensitivity  to  parameters. 

Quantification  of  the  relative  benefits  of  ensemble  size  versus  model  accuracy  have  been  derived  in 
the  special  case  of  Gaussian  uncertainties;  the  smallest  ensemble  size  likely  to  capture  "truth" 
within  the  bounding  box  of  the  ensemble,  has  been  computed  as  a  function  of  the  model  bias  and 
the  standard  deviation  of  the  perturbations  used  to  form  the  ensemble. 

We  have  begun  a  study  of  improvements  to  radial  basis  function  models  based  on  the  cylindrical 
basis  functions  of  Judd  (1998),  applications  to  simple  systems  show  promising  ensemble  forecasts; 
application  to  the  rotating  fluid  annulus  experiment  shall  begin  shortly. 

Alternative  methods  for  constructing  ensembles  have  been  explored  and  evaluated  using  our  new 
minimum  spanning  tree  test;  the  major  insight  gained  here  is  that  ensemble  forecasting  almost 
certainly  fails  to  give  accurate  probability  density  forecasts  unless  both  perfect  model  and  perfect 
ensemble  are  in  hand.  Thus,  rather  than  attempt  to  improve  ensemble  formation,  we  have 
concentrated  on  alternative  targets  for  ensemble  forecasting,  leading  us  to  the  bounding  box  and 
new  varieties  of  shadowing  trajectories. 

RESULTS 

Initial  studies  on  the  duration  of  the  linear  regime  in  operational  weather  forecasting  models 
revealed  that  the  regime  is  much  shorter  than  commonly  assumed,  implying  that  some  operational 
ensemble  formation  schemes  are  not  internally  consistent. 

It  is  almost  certainly  the  case  that  accurate  PDF  forecasts  of  the  weather  cannot  be  obtained; 
attempting  bounding  box  forecasts  is  shown  to  be  promising.  The  size  of  ensemble  required  to  form 
a  good  bounding  box  is  shown  to  remain  relatively  small,  even  in  high  dimensional  systems,  as 
long  as  the  model  error  (bias)  is  small;  and  to  increase  rapidly  in  even  moderate  dimensional 
systems  as  the  bias  increases. 


Minimal  Spanning  Trees  (MST)  have  been  used  to  generalise  the  Talagrand  diagram  to  higher 
dimensions,  thereby  allowing  more  robust  assessment  of  ensemble  forecasts. 


IMPACT/APPLICATIONS 


Initial  studies  suggest  that  the  eommonly  held  goals  of  ensemble  foreeasting  are  almost  eertainly 
unaehievable.  By  ehanging  the  questions  asked,  we  hope  to  allow  greater  use  of  the  foreeasts  for 
the  user. 

Future  work  should  allow  definitive  answers  on  distribution  of  eomputational  resourees  between 
improved  model  aeeuraey  and  ensemble  size. 

TRANSITIONS 

ECMWF  is  eonsidering  shortening  the  optimisation  time  in  its  operational  ensemble  foreeasts. 

RELATED  PROJECTS 

1-  Understanding  Model  Error.  A  graduate  studentship  sponsored  by  ECMWE;  David  Orrell  is 
studying  the  implieations  model  error  holds  for  ensemble  foreeasting  and  methods  to  seareh  for 
shadowing  orbits  in  operational  weather  models. 

2  -  Boiling  Dynamies.  Prof  David  Kenning  (Oxford,  engineering)  is  using  the  modelling  teehniques 
we  have  developed  to  study  the  dynamies  of  boiling. 

3  -  Roll  resonanee  and  intereept.  Torn  Hirose  (1999)  has  eompleted  a  Masters  Thesis  based  on  an 
applieation  of  our  work  on  data  assimilation  and  quantifying  uneertainty,  the  aim  of  the  projeet  was 
to  eonsider  the  likelihood  of  roll  resonanee  in  a  eylinder  falling  through  the  atmosphere  and  thereby 
eonfounding  simple  intereept  strategies. 

4  -  Rogue  thermoeouple  deteetion.  Eiam  Clarke  (1999)  has  eompleted  a  Masters  Thesis  applying 
our  approaeh  to  the  identifieation  of  intermittently  unreliable  sensors  in  a  long  data  stream 
eontaining  roughly  100  sensors.  The  industrial  applieation  was  for  monitoring  thermoeouples  in  the 
heat-exehangers  of  a  nuelear  power  plant.  British  Energy  (formally  Nuelear  Eleetrie)  has  funded  a 
D  Phil  studentship  whieh  Eiam  Clarke  has  taken  up  in  my  group. 
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